The disclosed embodiments relate generally to image processing. More specifically, the disclosed embodiments relate to techniques for registering two or more images so that they may be combined or “stitched” together. By way of example, the disclosed embodiment id directed to providing real-time panoramic photography image registration for handheld personal electronic devices with image sensors.
Panoramic photography may be defined generally as a photographic technique for capturing images with elongated fields of view. An image showing a field of view approximating, or greater than, that of the human eye, e.g., about 120° wide by 75° high, may be termed “panoramic.” Thus, panoramic images generally (but not always) have an aspect ratio of 2:1 or larger, meaning that the image is at least twice as wide as it is high (or, conversely, twice as high as it is wide, in the case of vertical panoramic images). In some embodiments, panoramic images may even cover fields of view of up to 360 degrees, i.e., a “full rotation” panoramic image.
There are many challenges associated with taking visually appealing panoramic images. These challenges include photographic problems such as: difficulty in determining appropriate exposure settings caused by differences in lighting conditions across the panoramic scene; blurring across the seams of images caused by the motion of objects within the panoramic scene; and parallax problems, i.e., problems caused by the apparent displacement or difference in the apparent position of an object in the panoramic scene in consecutive captured images due to rotation of the camera about an axis other than its center of perspective (COP). The COP may be thought of as the point where the lines of sight viewed by the camera converge. The COP is also sometimes referred to as the “entrance pupil.” Depending on the camera's lens design, the entrance pupil location on the optical axis of the camera may be behind, within, or even in front of the lens system. It usually requires some amount of pre-capture experimentation, as well as the use of a rotatable tripod arrangement with a camera sliding assembly to ensure that a camera is rotated about its COP during the capture of a panoramic scene. This type of preparation and calculation is not desirable in the world of handheld, personal electronic devices and ad-hoc panoramic image capturing.
Other challenges associated with taking visually appealing panoramic images include post-processing problems such as: properly aligning the various images used to construct the overall panoramic image (referred to herein as “image registration”); blending between the overlapping regions of various images used to construct the overall panoramic image; choosing an image projection correction (e.g., rectangular, cylindrical, Mercator) that does not distort photographically important parts of the panoramic photograph; and correcting for perspective changes between subsequently captured images.
Heretofore, panoramic photography systems have been unable to provide meaningful real-time or near-real-time panoramic image registration on full resolution image data captured during a panoramic sweep. This has been due, at least in part, to the onerous processing burden imposed by the feature detection and cross-correlation algorithms typically used to register the full resolution captured panoramic image portions.
Accordingly, there is a need for techniques to improve image registration performance during the capture and processing of full resolution (or otherwise high resolution) photographs, especially in the context of panoramic image processing on handheld, personal electronic devices such as mobile phones, personal data assistants (PDAs), portable music players, digital cameras, as well as laptop and tablet computer systems. By employing an intelligent sliding registration window scheme, and optionally leveraging a maximized energy function and feature caching, more effective panoramic photography image registration techniques, such as those described herein, may be employed to achieve real-time or near-real-time image registration of full resolution panoramic images.